A general framework for comparing (approximate) inference algorithms

Conclusive comparisons of approximate inference algorithms are difficult due a number of handicaps. The main problem is that most algorithms were developed independently for different domains and consequently have different input and output representations and formulations, for example, Bayesian networks vs Markov random fields (MRFs) [13]. Thus, researchers usually only compare novel approximate inference algorithms against a small subset of well-known algorithms instead of the current state-of-the-art. Analogously, practitioners usually employ well-known algorithms developed for their problem at hand, ignoring potentially superior algorithms that only require a slightly modified problem formulation.